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|t  was  shown  that  PCI  Is  related  to  Cliff's  consistency  Index  C  In  the 
following  manner.  When  the  items  are  arranged  In  ascending  order  of  the 
proportions  of  individuals  in  a  group  who  pass  the  items,  and  the  PCI  is 
computed  for  each  Individual,  a  certain  weighted  average  of  these  PCI's 
yields  an  index  which  is  a  slight  modification  of  Cliff's  C. 

One  way  of  conceptualizing  what  the  PCI  measures  is  the  extent  to  which 
each  individual's  particular  response  pattern  contributes  to,  or  detracts 
from,  the  overall  consistency  found  in  the  group's  mode  of  responding. 

The  foregoing  observations  make  it  clear  that  one  use  of  the  PCI  con¬ 
sists  in  spotting  anomalous  response  patterns  that  result  from  a  student's 
problems,  for  example.  From  this  It  is  a  short  step  to  utilizing  the  PCI 
for  identifying  a  subgroup  of  students  for  whom  the  given  set  of  items 
approximately  constitutes  a  unidimensional ly  scalable  set.  The  duality 
between  students  and  Items  then  permits  selection  of  a  subset  of  items  for 
further  improving  the  unidimensionality. 

Very  roughly  speaking,  the  culling  out  of  students  and/or  items  to 
achieve  unidimensionality  by  using  the  PCI  proceeds  as  follows.  Students 
whose  response  pattern  are  so  anomalous  (i.e.,  so  atypical  of  the  group)  as 
to  have  negative  PCI  values  are  eliminated  from  the  outset.  The  weighted 
average  of  the  PCI's  of  the  remaining  members  of  the  group  (referred  to 
above  as  resembling  Cliff's  C)  is  computed.  Then,  In  a  manner  somewhat 
analogous  to  removing  variables  in  the  backward  elimination  method  of 
stepwise  multiple  regression,  students  are  successively  removed  from  the 
group  in  such  a  way  that  at  each  step  the  PC  I -weighted-average  for  the 
remaining  group  shows  the  largest  increment  from  the  previous  value.  A 
suitable  stopping  rule  terminates  the  process  before  the  group  gets 
intolerably  small  In  size.  A  computer  routine  for  effecting  this  procedure 
was  developed. 

The  PCI  measures  the  degree  to  which  an  individual's  response  pattern 
resembles  the  group's  modal  response  pattern.  Sometimes,  however,  we  need 
a  measure  of  how  constant  an  Individual's  response  pattern  remains  for 
parallel  subsets  of  items  ocuurring  earlier  and  later  in  a  test.  One 
reason  for  this  is  that  students  often  switch  their  rules  of  operation  — 
either  from  one  erroneous  rule  to  another  or  from  an  erroneous  to  the 
correct  rule  —  as  they  proceed  through  a  test.  Thus,  their  response 
patterns  tend  to  be  Inconsistent  among  one  another  while  learning  is  taking 
place,  but  become  more  and  more  consistent  as  they  reach  mastery  level.  An 
index  for  measuring  individual  consistency  was  developed  and  called  the 
Individualized  Consistency  Index  (ICI), 
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ABSTRACT 

An  index  measuring  the  degree  to  which  a  binary  response  pattern 
conforms  to  some  baseline  pattern  was  defined  and  named  the  Pattern 
Conformity  Index  CPCI).  By  "baseline  pattern"  we  mean  a  binary  response 
vector  with  ail  the  O's  preceding  the  1's  when  the  items  are  arranged 
in  descending  order  of  difficulty  or  in  some  other,  purposefully  defined 
order. 

It  was  shown  the  PCI  is  related  to  Cliff's  consistency  index  C 
in  the  following  manner.  When  the  items  are  arranged  in  ascending  order 
of  the  proportions  of  individuals  In  a  group  who  pass  the  items,  and 
the  PCI  is  computed  for  each  individual,  a  certain  weighted  average 
of  these  PCI's  yeilds  an  index  which  is  a  slight  modification  of 
Cliff's  C. 

One  way  of  conceptualizing  what  the  PCI  measures  Is  the  extent 
to  which  each  individual's  particular  response  pattern  contributes  to, 
or  detracts  from,  the  overall  consistency  found  in  the  group  s  mode  of 
responding. 

The  foregoing  observations  make  it  clear  that  one  use  of  the  PCI 
consists  of  spotting  anomalous  response  patterns  that  result  from 
a  student's  problems,  for  example.  From  this  it  is  a  short  step  to 
utilizing  the  PCI  for  Identifying  a  subgroup  of  students  for  whom  the 
given  set  of  Items  approximately  constitutes  a  unidimensional ly 
scalable  set.  The  duality  between  students  and  Items  then  permits 
selection  of  a  subset  of  items  for  further  improving  the  unidimension¬ 
ality. 


Very  roughly  speaking,  the  culling  out  of  students  and/or  items 
to  achieve  unidimensionality  by  using  the  PCI  proceeds  as  follows.  Stu¬ 
dents  whose  response  pattern  are  so  anaomalous  (i.e.,  so  atypical  of 
the  group)  as  to  have  negative  PCI  values  are  eliminated  from  the 
outset.  The  weighted  average  of  the  PCI's  of  the  remaining  members  of 
the  group  Creferred  to  above  as  resembling  Cliff's  C)  is  computed.  Then, 
in  a  manner  somewhat  analogous  to  removing  variables  in  the  backward 
elimination  method  of  stepwise  multiple  regression,  students  are  successively 
removed  from  the  group  in  such  a  way  that  at  each  step  the  PCI -weighted- 
average  for  the  remaining  group  shows  the  largest  increment  from  the 
previous  value.  A  suitable  stopping  rule  terminates  the  process 
before  the  group  gets  intolerably  small  in  size.  A  computer  routine  for 
effecting  this  procudre  was  developed. 

The  PCI  measures  the  degree  to  hich  an  individual's  response 
pattern  resembles  the  group's  modal  response  pattern.  Sometimes,  however, 
we  need  a  measure  of  how  constant  an  individual's  response  pattern 
remains  for  parallel  subsets  of  items  ocurring  earlier  and  later  in  a 
test.  One  reason  for  this  is  that  students  often  switch  their  rules 
of  operation  —  either  from  one  erroneous  rule  to  another  of  from  an 
erroneous  to  the  correct  rule  —  as  they  proceed  through  a  test.  Thus, 
their  response  patterns  tend  to  be  inconsistent  among  one  another  while 
learning  is  taking  place,  but  become  more  and  more  consistent  as  they 
reach  mastery  level.  An  Index  for  measuring  Individual  consistency 
was  developed  and  called  the  Individualized  Consistency  Index  (ICI). 


DETECTION  OF  ABERRANT  RESPONSE 
PATTERNS  AND  THEIR  EFFECT  ON  DIHENSIONALITY 


KIkuinl  Tatsuoka  &  Maurice  Tatsuoka 

INTRODUCTION 

It  would  seem  trite  to  say  that  the  possiblity  of  an 
examinee's  getting  correct  answers  for  the  wrong  reasons  always  lurks 
behind  dichotomously  scored  test  items  and  threatens  to  destroy  the 
validity  of  the  test  <—  except  for  the  fact  that,  until  recently,  this 
possibility  has  largely  been  ignored  by  psychometricians.  Although 
scattered  attempts  have  been  made  to  give  partial  credit  for  partial 
knowledge,  procedures  for  discrediting  correct  answers  arrived  at  by 
incorrect  means  have  largely  been  confined  to  the  use  of  formulas  for 
correction  for  guessing.  This  paucity  may  not  be  devastating  for 
standardized  ability  tests,  but  is  practically  fatal  In  the  context  of 
achievement  testing  that  is  an  integral  part  of  the  Instructional 
process.  There  the  test  must  serve  the  purpose  of  diagnosing  what  type 
of  misconception  exists,  so  that  appropriate  remedial  instruction  may 
be  given.  This  calls  for  delving  into  the  cognitive  processes  that  are 
brought  Into  play  in  solving  problems,  and  trying  to  pinpoint  Just 
where  the  examinee  went  astray,  even  when  the  correct  answer  was 
fortuitously  produced. 

This  type  of  testing  was  pioneered  by  Brown  &  Burton  (1978), 
whose  celebrated  BUGGY  is  essentially  an  adaptive  diagnostic  testing 
system  which  utilizes  network  theory  for  routing  examinees  through  a 
set  of  problems  In  the  addition  and  subtraction  of  positive  Integers. 
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The  branching  Is  such  that  each  successive  problem  serves  to  narrow 
down  the  scope  of  "hypotheses"  as  to  the  type(s)  of  misconception  held 
by  the  examinee  until  finally  a  unique  diagnosis  Is  made. 

Tatsuoka  e^  al_.  (1980)  developed  a  diagnostic  testing  system 
which  differed  from  BUGGY  in  that  the  test  was  not  adaptive  but 
"conventional",  I.e.,  linear.  The  test  was  constructed  for  use  in 
conjunction  with  lessons  In  the  addition  and  subtraction  of  signed 
numbers  (positive  and  negative  integers)  for  eighth  grade  students 
and  consisted  of  four  parallel  subtests  of  16  items  each.  A  system  of 
error  vectors  was  developed  for  diagnosing  the  type(s)  of  error 
committed. 

Crucial  to  this  system  of  error  diagnosis  Is  the  ability  to 
tell  whether  and  to  what  extent  a  response  pattern  Is  "typical"  or 
"consistent".  We  may  speak  of  consistency  with  respect  to  either 
the  average  response  pattern  of  a  group  or  an  Individual's  own  response 
pattern  over  time.  To  measure  consistency  in  these  two  senses^,  two 
related  but  distinct  Indices  are  developed  in  this  paper.  They  are 
called  the  "Norm  Conformity  Index"  (NCI)  and  "Individual  Consistency 
Index"  (ICI),  respectively. 

It  Is  shown  that  a  certain  weighted  average  of  the  NCI's  of 
the  members  of  a  group  yields  one  of  Cliff's  (1977)  group  consistency 
indices,  C^.|.  The  higher  the  value  of  C^^,  the  closer  the  group  data 
set  Is  to  being  unidimenslonai  in  the  sense  of  forming  a  Guttman  scale, 
i^esponse  patterns  produced  by  erroneous  rules  are  usually  quite 
different  from  the  average  response  pattern.  Hence,  removing  individuals 
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wttK  low  (usually  negative)  NCI  values  —  I.e.,  those  with  aberrant 

<• 

response  patterns  —  will  yield  a  data  set  that  is  more  nearly 
«  unidimensional. 

The  ICI,  on  the  other  hand,  measures  the  degree  to  which  an 
Individual's  response  pattern  remains  invariant  over  time.  Thus,  for 
example,  in  the  signed-number  test  consisting  of  four  parallel 
subtests,  the  ICI  indicates  whether  an  examinee's  response  pattern 
changes  markedly  from  one  subset  to  the  next  or  remains  relatively 
stable.  Low  ICI  values,  indicating  instability  of  response  pattern, 
would  suggest  that  the  examinee  was  still  in  the  early  stages  of 
.  learning,  changing  his/her  method  for  solving  equivalent  problems 

from  one  wave  to  the  next.  A  high  ICI  value,  reflecting  stability 
of  response  pattern,  would  signal  the  nearing  of  mastery  or  a  learning 
plateau. 

While  the  NCI  and  ICI  can  each  serve  useful  purposes  as 
suggested  above  and  illustrated  in  detail  below,  examining  them  Jointly 
opens  up  various  diagnostic  possibilities,  as  does  the  consideration  of 
each  of  them  in  combination  with  the  total  test  score. 
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NORM  CONFORMITY  INDEX 

Cliff  (1977)  defined  various  consistency  indices  based  on  the 
notion  of  dominance  and  counterdominance  relationships  between  pairs 
of  items.  Some  of  these  are  closely  related  to  indices  developed 
in  the  theory  of  scalability,  originating  in  the  i930's.  Although 
Cliff's  Indices  are  derived  from  the  dominance  matrix  for  the  data 
set  of  an  entire  group,  they  can  be  expressed  as  weighted  averages 
of  the  corresponding  indices  based  on  constituent  subgroups  of 
examinees.  (Krus,  1975;  Mokken,  1970;  Yamamoto  &  Wise,  I98O.) 
Nevertheless,  It  should  be  noted  that  these  indices  are  measures  of 
group  consistency  and  do  not  represent  individual  examinees'  consistency 
of  responses. 

Birenbaum  &  Tatsuoka  (1980)  demonstrated  that  individual  response 
patterns  offer  powerful  information  for  determining  any  erroneous 
rule  of  operation  that  a  given  examinee  may  have  used  in  taking  a 
test.  Tatsuoka  et  £]_.  (I98O)  developed  a  diagnostic  system  for 
identifying  erroneous  rules  by  generating  "error  vectors",  each  of 
whose  binary  elements  represents  the  presence/absence  of  a  specific 
"atomic"  error.  In  this  paper  we  develop  an  index  that  associates  with 
each  response- pat tern  vector  a  number  between  -I  and  I  (inclusive) 
representing  the  degree  of  concordance  the  vector  shows  with  a  Guttman 
vector  of  the  same  length  (I.e.,  the  same  number  of  I's)  with  the 
items  arranged  in  some  purposefully  specified  order.  For  instance, 
they  may  be  arranged  —  as  they  are  in  computing  Cliff's  indices  — 
in  descending  order  of  difficulty  for  the  total  group;  or  they 
may  be  arranged  in  any  particular  order  that  suits  a  given  purpose. 
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It  should  be  noted  that  group  consistency  in  Cliff's  sense  is 
maximized  when  the  Items  are  ordered  by  difficulty.  Any  change  of 
item  ordering  would  result  in  a  decrease  in  the  value  of  any  of 
Cliff's  consistency  indices.  The  value  yielded  by  each  of  Cliff's 
formulas  may,  therefore,  be  regarded  as  a  function  of  item  order. 

Consider  a  dataset  consisting  of  just  one  person's  response- 
pattern  row-vector  S.  The  dominance  matrix  for  this  response  pattern 
is 

(1)  S'S  =  N  =  ®  1,2,. ..,n  (“  number  of  items) 

where  S'  is  the  transpose  of  the  complement  of  S.  By  construction, 
rtjj  »  1  when  the  individual  gets  item  I  wrong  and  Item  j  right; 
otherwise  n^j  =  0. 

Of  course  if  the  ordering  of  the  items  in  S  is  changed,  the 
dominance  matrix  will  also  change.  Consequently,  the  consistency 
index  associated  with  response-pattern  S,  defined  as 

(2)  c„  =  2u yu  -  1  , 

P  ® 

where  U  ■  E  E  n..  (the  sum  of  the  above-di agonal  elements  of  N) , 
®  i  j>l  ‘J 

and  U  ■  E  E  n,.  (the  sum  of  all  the  elements  of  N) , 

J  i 

is  a  function  of  the  item  order,  0.  To  make  this  fact  explicit, 
we  write  C  (0) . 
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Example  1 ;  Let  S  ■  (10110);  then 


1 

N  -  S'S  0  (10110) 

0 
I 


fo  0  0  0  0  ' 

i 

I  1  0  1  10 


0  0  0  0  0; 
0  0  0  0  0  I 
I  0  I  I  Oj 


Here  U  •  Z  E  n,.  ■  2  and  U  *  6  ;  hence  from  Equation  (2), 
^  I  j>l 


Cp(0)  -  (2)(2)/6  -  1  -  -1/3  . 


Example  2;  Let  S  •  (00111),  the  Guttman  Vector  with  three  1'$. 

Then 


'  1  ' 

I'o  0 1 1  r 

1 

i  0  0  ri  1 

0 

(00111)  - 

0  0  0  0  0 

0 

0  0  0  0  0 

.  0  . 

0  0  0  0  0 

-  6,  U  -  6  ;  hence  C„(0)  -  12/6  -  1  -  1  . 
a  p 
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Example  3:  Let  S  ■  (1 1 100),  the  "reversed  Guttman  vector".  Then 


'o  0  0  0  o' 

i  0  1 

0  0  0  0  0 

0  :(iiioo)  = 

t 

0  0  0  0  0 

1 

1 

1110  0 

1  1  10  0 

Here  -  0,  U  -  6;  hence  C  (O)  =0/6-1  -1  . 

a  p 

From  the  foregoing  examples,  the  first  two  of  the  following 
properties  of  Cp(0)  may  be  inferred.  The  other  properties  are 
illustrated  by  further  examples,  and  intuitive  arguments  are  given 
to  substantiate  them.  Their  formal  proofs  are  not  difficult  but 
tedious,  and  are  therefore  omitted. 

Property  1 :  -1  s  0^(0)  s  1 

Property  2;  If  the  order  of  Items  is  reversed  in  S,  the 

absolute  value  of  C^(0)  remains  unchanged, 
but  its  sign  is  reversed. 

Since  U"rZn,,»EE  (1-s,)S.,  It  is  Invariant  with  respect 
J  I  J  I  ' 

to  permutations  of  the  elements  of  S.  On  the  other  hand,  if  the 
order  of  the  elements  of  S  is  reversed,  so  that  Sj  -  , 

the  U.  for  the  new  dominance  matrix  will  become  U'  ■  E  E  n', ,  ■ 

*  ^  i  j>I  ’J 

n  n 

E  E  ,  which  can  be  shown  to  be  equal  to  U  -  U.. 

{•]  J-l+1  n-I+I  n-J+i  a 
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Therefore  Cp(0)  ■  2(U-U^)/U  -  I 

-  -2U^/U  +  1  -  -Cp(0) 


Property  3:  The  consistency  Index  0^(0)  associated  with  i 
matrix,  comprising  two  response-pattern  vectors  and  $2 
weighted  average  of  the  0^(0) 's  associated  with  and  $2 


the  consistency  index  for  S  is 


Cp(0) 


(S-,  S'j) 


4 1- 


Therefore,  If  we  let 


U.  -  r  Z  n[^^  for  k  -  1,2 
K  j  I  'J 

and  Ul,  “  2  £  for  k-  1,2  , 

•«  ,  j5.j  IJ 


It  fol1o%«s  that  the  U  and  U_  for  S  are  given  by 

O 


and 


U  -  Uj  +  U2 


^a-‘'la*«2a  * 


Hence, 


Cp(0) 


^<»la  ^  "2a> 

Ui  +U2 


-  1 


2U 


U,  +U2 


la 


2U 


2a 


+U2 


-  1 


2  X  n  data 
Is  a 

respectively.  It 
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=  W,c  (0),  +  W2C  (0), 

1  p  I  P  2 


Remark;  The  two  response  patterns  and  may  be  either  those  of 

two  individuals  are  of  a  single  individual  taking  a  set  of  items  on 

two  occasions  C^s  in  a  repeated-measures  design)  or  two  parallel 

sets  of  items.  In  the  first  case  the  C  (0)  associated  with  the  2xn 

P 

data  matrix  would  be  an  average  C^(0}  for  the  pair  of  individuals; 
in  the  second,  it  would  be  an  average  over  two  measurement  occasions 
for  one  person. 

By  mathematical  induction  on  Property  3»  it  follows  that  the  C^lO) 
associated  with  an  N  x  n  data  matrix 


X  - 

is  a  weighted  average  of  the  C  (0)'s  associated  with  the  individuai 

P 

response-pattern  vectors  Sj,  $2,  ...  ,  S|^.  In  particular,  when  the 
items  are  arranged  in  descending  order  of  difficulty  for  the  group 
comprising  the  N  individuals,  the  €^(0)  associated  with  X  is  one  of 
Cliff's  CI977)  consistency  Indices,  For  this  particular  ordering 

of  Items,  we  give  the  name  "norm  conformity  Index"  to  the  CpC0)*s 
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associated  with  the  Individual  response  patterns. 

Definition;  Norm  Conformity  index.  NCI 

When  the  Item  ordering  is  in  descending  order  of  difficulty  for 
a  particular  group  (designated  the  "norm  group"),  the  consistency  index 
Cp(0}  associated  with  the  individual's  response  pattern  S  is  called 
the  norm  conformity  index,  denoted  by  NCI.  Thus,  NCI  indicates  the 
extent  to  which  a  response  vector  S  approximates  the  Guttman  vector 
(In  which  all  the  zeros  are  to  the  left  of  the  I's)  with  the  same 
number  of  I's,  when  the  Items  are  arranged  In  descending  order  of 
difficulty  for  the  norm  group. 

With  this  definition,  plus  an  expanded  version  of  Property  3> 
we  state  the  relationship  between  Cliff's  consistency  index  C^j  and 
the  NCI's  for  the  individuals  in  the  group  as 

Property  ki  Cliff's  consistency  index  C^^  is  a  weighted  average 
of  the  NCI|^  (k-1,  2 . N),  with  weights  Wj^  ■ 

N 

C.,  -  Z  (U./U)  NCI., 
ti  1^.,  K  K 


where 


and 


Z 

I 


z 

i>\ 


n 


(k) 

U 


N 

k-1 


Example  k\  Let  S,  -  (01011),  -  (00111)  and  -  (00001)  be 

the  response-pattern  vectors  for  three  Individuals.  Then,  by 
calculations  similar  to  those  shown  in  Examples  I,  2,  and  3,  we  get 
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(upon  writing  NCI  for  0^(0)) 

Ufa  -  5.  U,  -  6,  NCI,  -  2/3 

^23  -  6,  U2  -  6.  NCI2  -  I 

U^a  •  U3  -  4,  NCI3  -  1 

Hence, 

WjNCI,  +  W2NCI2  +  W3NCI3  -  (6/l6){2/3)  +  (6/l6)(l)  +  (4/l6)(1) 

*  7/8  . 

On  the  other  hand,  with 

‘0101  1 

X  -  0  0  0  0  1 

0  0  111 

■  0  1  123 
0  0  112 
X'X  »  N  -  01012 

0  0  0  0  1 
0  0  0  0  0 

-  15,  u  -  16,  NCI  -  30/16  -  1  -  7/8, 

Q 

thus  verifying  Property  4. 

In  the  paragraph  preceding  Property  4,  the  order  of  the  items 
was  taken  to  be  the  order  of  difficulty  for  the  group  of  which 
the  Individual  was  a  member,  for  0^(0)  to  be  called  NCI.  Actually, 
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as  evident  in  the  formal  definition  of  NCI»  the  group  need  not  be 
one  to  which  the  Individual  belongs.  It  can  be  any  group  which  the 
researcher  chooses  for  defining  the  baseline  or  "criterion  order" 
of  the  items;  hence  our  referring  to  it  as  the  norm  group,  and  the 
index  as  the  norm  conformity  index.  Thus,  for  example,  we  might  be 
concerned  with  two  groups  of  students  with  vastly  different  instruc¬ 
tional  backgrounds  but  similar  abilities.  It  is  then  quite  possible 
for  the  difficulties  of  various  skills  to  be  rather  different  in  the 
two  groups.  We  might  take  Group  I  as  the  norm  group,  thus  arranging  the 
items  in  descending  order  of  difficulty  for  this  group.  We  could 
compute  NCI's  for  members  of  both  Group  1  and  Group  2  on  the  basis  of 
this  criterion  order,  and  would  probably  find  the  mean  NCI  for  the 
two  groups  to  be  significantly  different.  The  following  examples, 
based  on  real  data,  illustrate  this. 

Example  5i  The  seventh  grade  students  of  a  junior  high  school 
were  divided  at  random  into  two  groups,  which  were  given  different 
lessons  teaching  signed-number  operations.  (Tatsuoka  &  BIrenbaum, 

1979).  One  sequence  of  lessons  taught  the  operations  by  the  Postman 
Stories  approach  (Davis,  19^4)  while  the  other  used  the  number- line 
method. 

After  addition  problems  had  been  taught,  a  52-item  test  including 
both  addition  and  subtraction  problems  was  administered  to  ail  students. 

A  t-test  showed  no  significant  difference  between  the  mean  test 
score  of  the  two  groups,  as  indicated  in  Table  1.  However,  when  NCI's 
were  computed  for  all  students,  using  the  item-difficulty  order  in 


Group  I  (the  Postman** Stories  group)  as  the  baseline,  there  was  a 
significant  difference  between  the  mean  NCI  of  the  two  groups. 

Insert  Table  1  about  here 

The  means  of  test  scores  and  the  Norm  Conformity  Index 


HSHEREE3I 

Total  Score 

mean 

20.06 

18.36 

t  -  1.190 

SO 

8.30 

7.88 

P  >  .05 

NCI 

mean 

.55 

.45 

t  -  2.246 

SO 

.23 

.23 

p  -  .0264 

Example  6;  Tatsuoka  &  Birenbaum  (1979)  demonstrated  that 
proactive  inhibition  affected  the  performance  on  tests  In  material  learned 
through  subsequent  instructions.  The  response  patterns  of  students  who 
studied  new  lessons  written  by  using  a  different  conceptual  framework 
from  that  of  their  previous  instructions  showed  a  significantly  different 
performance  pattern.  By  a  cluster  analysis,  four  groups  among  which 
response  patterns  are  significantly  different  were  Identified.  The  NCI 
values  for  91  students  based  on  the  order  of  tasks  determined  by  the 
proportion  correct  In  the  total  sample  were  calculated  and  analysis  of 
variance  was  carried  out.  The  F-value  was  significant  at  p  ■  0.05. 


Insert  Table  2  about  here 
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Table  2 

ANOVA  of  Norm  Conformity  Index  for  Four  Groups 
With  Different  Instructional  Backgrounds 


Group 

N 

Mean  of  NCIs 

F 

1 

34 

0.18 

3.62  with  df  -  3,  87 

2 

27 

0.41 

3 

20 

0.35 

4 

10 

0.18 

Up  to  this  point,  the  U  and  U  in  Eq.  (2)  defining  C  (0)  —  and 

a  p 

hence  NCI  as  a  special  case  were  defined  in  terms  of  the  numbers  of 
dominances  and  counter-dominances  between  item  pairs  in  the  dominance 
matrix  N.  We  now  show  that  U  can  be  explicitly  defined  in  terms  of 
the  proximity  of  a  response  vector  S  to  a  Guttman  vector  with  the  same 
number  of  I's. 

Property  5«  Let  S  be  a  response-pattern  vector  of  an  examinee  on  an 
n-item  test,  N  ■  S'S  the  associated  dominance  matrix,  and 

n 

U  -  r  ^  "it 

*  i-I  j>i  'J 

Then  is  also  the  number  of  transpositions  required  to  get  from  S  to 
the  reversed  Guttman  vector  (all  Is  preceding  the  zeros). 

Since  n|j  ■  (I  -  S|)Sj»  It  follows  that 

U  -  E  Z  (I  -  s,)s, 

*  I  J>l  '  J 
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fs  the  number  of  ordered  pairs  (S|,  Sj)  [i<j]  of  elements  of  S 
such  that  S|  ■  0  and  Sj  ■  1.  That  Is,  If  for  each  S|  ■  0,  we  count  the 
number  of  Sj  ■  1  to  Its  right  In  S,  then  the  sum  of  these  numbers  over 
the  set  of  O's  In  S  Is  equal  to  U  .  But  this  Is  the  same  as  the 
number  of  transpositions  (interchanges  of  elements  in  adjacent  (0,1) 
pairs)  needed  to  transform  S,  step  by  step,  into  (II  ...  I  0  0  ...  0). 
Thus  is  a  measure  of  remoteness  of  S  from  the  reversed  Guttman 
vector,  which  Is  equivalently  Its  proximity  to  the  Guttman  vector. 

Example  7S  Let  S  •  (  0  I  0  I  I  ),  Then,  S  can  be  transformed 
into  (  I  1  I  0  0  )  by  five  successive  transpositions: 

(CM  0  1  1)  (1  0  0  I  1)  -»•  (1  0  I  0  I) 

(I  I  0  OJ)  (1  1  0  I  0)  (I  1  1  0  0); 

thus  Ug  ■  5  by  the  present  definition.  On  the  other  hand. 


■  I  ‘ 

0  10  1  1  ‘ 

0 

0  0  0  0  0 

1 

o 

o 

1 — ( 

M 

0  1  0  1  1 

0 

0  0  0  0  0 

,  0  _ 

0  0  0  0  0 

•  ^ 

and  U  ■  Z  Z  n,.  ■  5  by  the  earlier  definition. 

*  I  j>l  '■> 

It  may  also  be  noted  that,  if  we  denote  the  number  of  1's  in 
the  lower  triangle  of  F'S  by  i.e., 

then  is  the  numberof  ordered  pairs  (sj,  Sj)  [j  <  l]  of  elements 
of  S  such  that  Sj  ■  0  and  Sj  ■  I.  Hence, 
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U  -  E  E  n, ,  -  u,  +  u. 

j  ,  IJ  a  b 

Is  the  number  of  pa  I rs  (sj,  Sj)  with  Sj  #  Sj  that  can  be  formed  from 
the  elements  of  S.  Thus,  U  ■  x(n  -  x),  where  x  Is  the  number  of  I's 
In  S,  or  the  test  score  earned  by  a  person  with  response  pattern  S. 
Consequently,  U^/U  and  U|^/U  are  the  proportions  of  (0,1)  pairs  and 
(1,0)  pairs,  respectively,  among  all  possible  ordered  pairs’ (s.,  Sj) 

^i  <  j]  of  unlike  elements.  When  S  is  a  Guttman  vector  (0  0  ...0  I  I 
»  U  and  -  0,  because  all  ordered  pairs  of  unlike  elements  are 
(0,1)  pairs.  Conversely,  when  S  is  a  reversed  Guttman  vector 
(11  ...  100«..0),U=0  and  U.  ■  U.  Hence  U  /U  ranges  from 

o  D  o 

0  to  1  as  an  increasing  function  of  the  degree  to  which  S  resembles 
(or  is  proximal  to)  a  Guttman  vector.  Similarly,  U^/U  measures 
the  proximity  of  S  to  a  reverse  Guttman  vector,  or  its  remoteness 
from  a  Guttman  vector.  In  fact  U  /U  was  denoted  by  U  and  proposed 
as  an  index  of  "deviance"  of  score  patterns  by  van  der  Flier  (1977). 
With  the  above  redefinition  of  U  and  U,  the  sense  in  which  NCI 

O 

Is  a  measure  of  the  extent  to  which  a  response  pattern  approximates 
a  Guttman  vector  should  have  become  clearer. 


NCI  -  2U^/U  -  i 

O 

is  a  rescaling  of  U./U  to  have  limits  1  and  -I  Instead  of  1  and  0. 

a 

It  should  be  noted  that  U^/U,  and  hence  also  NCI,  is  undefined 
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for  a  person  who  has  a  test  score  of  either  0  or  n,  since  U  >  x(n  '  x) 

>  0  in  both  these  cases.  There  are  two  ways  (at  least)  in  which  to  cope 
with  this  problem*  The  first  is  arbitrarily  to  set  NCI  >■  I  when  U  >  0, 
which  is  analogous  to  setting  O!  >  I.  This  is  reasonable  because 
U  ■  0  only  for  S  ■  (0  0  ...  0)  and  S  =  (I  I  ...  I),  both  of  which 
are  Guttman  vectors  in  the  sense  of  having  no  zero  to  the  right  of 
any  I.  The  second  solution  is  to  redefine  NCI  itself  as 

(3)  NCI  -  2(Ug  +  l)/(U  +  I)  -  I  , 

which  will  automatically  make  NCI  >  I  for  the  all-correct  and 
al I- incorrect  response  patterns.  Each  of  these  solutions,  however, 
gives  rise  to  problems  of  its  own,  as  shown  in  the  discussion  section 
below. 

Property  6!  Suppose  Sj  and  $2  are  two  n- 1  tern  response  patterns 
withthe  same  number  x  of  I's,  and  that  $2  results  from  by 
applying  t  successive  transpositions.  Then 

Cp(S2)  “  Cp(S,)  ±  2t/x(n  -  x)  , 

where  the  +  sign  is  taken  when  $2  is  closer  to  a  Guttman  vector  than 
is  Sj  and  the  -  sign  when  the  opposite  Is  true.  # 

From  Property  5  the  U  associated  with  a  given  response  pattern 

9 

S  Is  the  number  of  transpositions  necessary  for  getting  from  S  to  the 
reversed  Guttman  vector  with  the  same  number  of  I's.  Hence,  if  t 
Is  the  number  of  transpositions  It  takes  to  get  from  Sj  to  $2,  it 
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fol lows  that 


if  $2  is  farther  from  the  reversed  Guttroan  vector,  i.e.,  closer  to 
the  Guttman  vector,  than  is  S^,  and 


U 


2a 


t 


if  the  opposite  is  true.  Consequently, 


2U 
=  U 


2a 


2(Uia  +  t) 


— -  I  + 

u  '  u 


* 


2t 


r(n^xT 


i 


when  $2  is  closer  than  S|  to  the  Guttman  vector.  The  sign  preceding 
2t/x(n-x)  becomes  -  when  $2  Is  farther  than  Sj  to  the  Guttman  vector. 

Example  8;  Let  Sj  »  (i  0  i  0  i  1)  and  $2  =  (0  i  0  i  i  i). 

It  takes  two  transpositions  to  get  from  to  $2  : 

(i  0  i  0  i  i)  (0  i  J_£  i  i)  (0  i  0  i  i  i), 

and  $2  Is  closer  than  Sj  to  the  Guttman  vector  (0  0  I  I  I  I).  Therefore, 
by  Property  6  we  should  have 


Cp(S2)  -  Cp(S,)  +  (2)(2)/(i»)(2) 

-  Cp(S,)  +  i/2  . 


i 
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For  the  two  response  patterns,  we  have 
Ula  "  5  and  ~  7, 

so  that 

Cp(S,)  -  (2)(5)/8-1  -  1A 

and 

C  (Sj  -  (2)(7)/8  -  1  =  3A 
P  2 

satisfying  the  above  relation. 


Property  7*  The  weights  applied  to  individual  NCI's  in 
computing  Ciiff's  consistency  index  (cf.  Property  k)  are  invariant 
of  changes  in  the  basei  ine  order  of  items.  It 


This  is  true  because  the  weights 


depend  only  on  U  “  x  (n  -  s  ),  where  x  is  the  total  score  earned 
P  P  P  P 

by  person  p  —  i.e.,  on  the  number  of  I's  in  response  pattern  Sp,  and 
not  on  their  positions. 

It  follows  that  NCI's  associated  with  response  patterns 
yielding  scores  close  to  n/2  get  high  weights  while  those  corresponding 
to  extreme  scores  get  low  weights.  It  is  also  seen  that  when  the  number 
of  persons  is  large,  each  Wp  is  a  fairly  small  positive  number,  while 
the  NCI  has  a  value  between  1  and  -I.  Negative  NCI's  are  an  obstruction 
to  having  a  large  group  consistency  index,  C^^. 
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INDIVIDUAL  CONSISTENCY  INDEX 


In  the  preceding  section  we  defined,  and  described  various 
properties  of,  an  index  which  measures  the  extent  to  which  an 
individual's  response  pattern  "conforms"  to  that  of  a  norm  group. 

In  some  situations  it  is  desirable  to  measure  the  extent  to  which  an 
individual's  response  pattern  remains  unchanged  or  "consistent"  over 
the  passage  of  time.  For  example,  it  is  reasonable  to  expect  that, 
when  a  student  is  in  the  process  of  learning  --  and  hence  presumably 
modifying  the  cognitive  processes  by  which  he/she  attempts  to  solve 
problems  —  his/her  pattern  of  responses  on  successive  sets  of  similar 
items  will  change  considerably  from  one  set  to  the  next.  When  the 
student  approaches  mastery  or  a  "learning  plateau",  his/her  response 
pattern  will  probably  remain  relatively  consistent  from  one  set  to 
the  next.  To  define  an  index,  called  the  Individual  Consistency  Index 
(I Cl),  that  will  serve  to  measure  the  degree  of  consistency  (or 
stability)  of  an  individual's  response  pattern  over  time,  and  to 
investigate  its  properties,  are  the  purposes  of  this  section.  In  the 
interest  of  clarity  and  ease  of  exposition,  we  embed  our  discussions 
in  the  context  of  an  actual  experimental  study. 

A  64- item,  signed-number  test  was  administered  to  153  seventh 
graders  at  a  Junior  high  school.  The  test  comprised  16  different 
tasks  being  tested  by  four  parallel  items  each.  The  items  were  arranged 
so  that  four  parallel  subtests  were  successively  given  to  each  testee. 
Within  each  16-item  subtest,  the  order  of  items  was  randomized.  Thus, 
fot  each  examinee  there  are  four  response-pattern  vectors  with  16 


elements  each.  The  Individual  Consistency  Index  (ICI)  is  defined  on 
these  four  replications.  We  shall  come  back  to  this  test  later,  but  we 
first  Introduce  ICI  by  a  simpler  example.  Suppose  a  person  took  four 
parallel  tests  A,  B,  C,  D  with  seven  items  each,  and  that  his/her 
response  patterns  were  as  shown  in  the  second  coiumn  of  Tabie  3  •  Also 
shown  in  this  table  are  U  =>  x(7  ~  x)  for  each  response  pattern,  the 
number  U  of  transpositions  needed  to  transform  each  response  pattern 

Q 

into  a  reverse  Guttman  vector,  the  C^(0)  for  each  response  pattern,  and 
the  weight  to  be  appiied  to  each  C^(0)  for  getting  an  overail  index. 


Table  3 

Four  response  patterns  and  various  quantities 
associated  with  them. 


para  1  lei 
test  #(]) 

Response 

Pattern 

U. 

J 

‘'ja 

1 

(lOlOOlO) 

12 

4 

-.333 

.286 

2 

(OOlOOlO) 

10 

6 

.200 

.238 

3 

(10000 10) 

10 

4 

-.200 

00 

4 

(1 000010) 

10 

4 

-.200 

.238 

The  weighted  average 

4 


E  w,  C„(0),  -  -.143 


j-1 


J  P 


would  be  Cliff's  consistency  Index  If  the  four  response  patterns  of 
Table  2  were  those  of  four  Individuals  and  if  the  items  had  been  arranged 
in  their  difficulty  order  for  the  group.  Let  us  rearrange  the  Items  (or 
rather  the  sets  of  parallel  Items)  In  their  order  of  difficulty  for  the 


for  the  person,  which  is  (2, 4, 5, 7, 3,1 ,6) .  The  response  patterns  and 
other  quantities  occurring  in  Table  3  now  become  as  shown  in  Table  4, 


which  also  has  a  new  column  showing  the  number  of  transpositions  t. 

J 

necessary  to  get  from  the  jth  response  pattern  in  Table  3  to  the  new 

1 

one  here. 

Table  4 

Response  patterns  resulting  from  those  in 
Table  2  by  arranging  the  Items  in  difficulty 
order,  and  various  associated  quantities. 


iliel 

:  #  (j) 

Response 

Pattern 

t. 

J 

U. 

J 

U1 

ja 

C  (O'). 

P  J 

w. 

J 

1 

(OOOOill) 

8 

12 

12 

1.0 

.286 

2 

(OOOOiOl) 

3 

10 

9 

.8 

.238 

3 

(0000011) 

6 

10 

10 

l.O 

.238 

4 

(000001 1 ) 

6 

10 

10 

1.0 

.238 

Note  that  the  new  0^(0)  for  each  response  pattern  satisfies  Property  6; 


Cp(0')j  -  Cp(0).  ♦  2tj/Uj 


The  weighted  average  of  the  new  C  (0)  values  is 
4 

Z  w.C  (O'),  -  .9524  . 


j-1 


'J  P' 


This  is  what  we  call  the  Individual  Conformity  Index,  ICI.  We  may  state 
its  definition  as  follows. 


Definition;  Individual  Conformity  Index  (ICI).  Given  a  set  of 
response  patterns  shown  by  a  single  Individual  on  a  set  of  parallel 
tests,  we  arrange  the  parallel  items  in  their  overall  order  of  difficulty 
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for  the  individual  and  compute  the  0^(0)  for  each  response  pattern 
thus  modified.  If  we  now  form  a  weighted  average  of  these  0^(0) 's 
as  though  we  were  computing  Cliff's  in  accordance  with  Property  4, 
the  result  is  the  ICI. 

Remark;  Note  that  ICI  is  an  attribute  of  a  single  individual, 
not  of  a  group  as  is  Cliff's  consistency  index.  ICI  differs  also  from 
NCI  in  that  the  latter  (also  an  individual  attribute)  depends  on  the 
baseline  order  of  items,  i.e.,  the  difficulty  order  in  some  group 
specified  as  the  norm  group,  whereas  ICI  is  computed  for  an  individual 
with  no  reference  to  any  group.  Rather,  ICI  requires  that  the 
individual  in  question  has  taken  two  or  more  parallel  tests,  and 
measures  the  consistency  of  his/her  response  patterns  across  these 
parallel  tests. 

Property  8;  Since  the  parallel  Items  are  arranged  in  their 
order  of  difficulty  for  the  individual  In  question  when  ICI  is  computed, 
while  they  are  arranged  in  their  order  of  difficulty  for  a  norm  group 
when  NCI  is  computed,  it  follows  that 

ICI  i  NCI 

for  each  examinee. 
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APPLICATION  TO  ERROR  ANALYSIS:  I 

Btrenbaum  &  Tatsuoka  (1930)  found  that  1-0  scoring  based  simply 
on  right  or  wrong  answers  caused  serious  problems  when  erroneous  rules 
of  signed-number  operations  were  used  by  many  examinees.  The  point  Is 
that  many  erroneous  rules  can  lead  to  correct  answers  In  many  items. 

To  highlight  the  extent  of  the  problem,  Tatsuoka  e^  aj^.  (1930) 
developed  an  almost  exhaustive  set  of  72  erroneous  rules  for  doing 
addition  and  subtraction  of  pairs  of  signed  numbers,  and  enumerated  the 
number  of  correct  answers  that  would  result  from  consistently  using 
each  incorrect  rule  for  a  set  of  16  items.  The  resulting  histogram 
is  shown  in  Figure  1,  where  it  can  be  seen  that,  in  an  extreme  case, 

12  out  of  the  16  items  could  be  answered  correctly  by  an  erroneous 
rule  of  operation. 

Using  real  data  from  a  64-item  test  consisting  of  four  parallel 
subtests  of  16  items  each,  Birenbaum  &  Tatsuoka  first  did  a  principal 
components  analysis  on  the  original  data  —  with  the  items  scored  1  or 
0  In  the  usual  manner.  Next,  the  data  were  modified  by  giving  a  score 
of  0  when  an  Item  was  correctly  answered  presumably  by  use  of  an 
erroneous  rule,  and  another  principal  components  analysis  was  done. 
(Details  of  how  it  was  judged  that  a  correct  answer  was  arrived  at 
by  an  incorrect  rule  are  given  in  Birenbaum  &  Tatsuoka,  1930.)  The 
change  between  the  two  analyses  was  dramatic.  The  dimensionality  of 
the  data  became  much  more  clearcut  with  the  modified  data.  The 
item-total  correlations  became  much  higher,  while  the  means  of  the  16 
tasks  (each  represented  by  four  parallel  items)  did  not  change 
significantly. 


•  •  •  • 
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Figure  1.  Histogram  of  total  scores  generated  by 
erroneous  rules  of  operation. 
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The  above  phenomenon  suggests  why  some  achievement  tests  cannot  be 
treated  as  unidimensional  even  though  the  items  are  taken  from  a  single 


content  domain.  The  fact  that  correct  answers  can  be  obtained  by  erroneous 
rules  apparently  makes  for  a  chaotic,  multidimensional  domain  (Tatsuoka  & 
Birenbaum,  1979) >  which  is  "cleaned  up"  by  the  rescoring.  Brown  &  Burton 
(1978)  warned  of  the  same  problem,  namely  that  wrong  rules  can  yield  the 
correct  answers  in  some  items  involving  addition  and  subtraction  of 
positive  integers. 

The  indices  developed  in  this  paper,  NCI  and  ICI,  are  useful  for 
detecting  erroneous  rules  that  are  consistently  used  by  an  examinee  or  a 
group  of  examinees.  This  capability  is  useful  not  only  in  the  teaching 
process,  for  diagnosing  a  student's  problem,  but  also  gives  some  ieads  to 
addressing  some  psychometric  issues  such  as  the  dimensionality  of 
achievement  tests. 

Table  3  shows  a  2  x  2  contingency  table  based  on  combinations  of 
high  and  low  NCI  and  ICI  values,  with  a  characterization  of  the  status 
of  students  in  each  cell,  dependent  also  on  the  score  earned. 

Table  S 

Types  of  students  with  high  and  low  NCI,  ICI  and  score. 


Low 

High 

High 

There  should  be  few  students 
in  this  cell  (none  if  the 
cutting  points  for  ICI  and 

NCI  are  the  same,  since 

ICI  >  NCI  always) . 

If  score  Is  high,  all  is  well. 

If  score  is  low,  student  has  a 
serious  misconception  (consistently 
uses  an  incorrect  rule)  which,  how¬ 
ever,  leads  to  correct  answers  to 
easy  items  and  wrong  answers  to 
hard  Items. 

The  errors  are  probably 
random. 

If  score  is  high,  student  is  merely 
getting  a  few  of  the  easy  items 
wrong. 

Low 

If  score  is  low,  student  Is  getting 
many  of  the  easy  Items  wrong.  The 
response  pattern  is  strange,  and  a 
serious  problem  exists. 

a 


Example  9;  The  example  described  at  the  beginning  of  the  section  on 
the  Individual  Consistency  Index  was  thoroughly  analyzed  In  Technical 
Report  80-1  (Birenbaum  S  Tatsuoka,  I98O)  with  respect  to  error  analyses. 

We  call  this  data  the  November  data  hereafter.  There  are  16  different 
erroneous  rules  diagnosed  In  the  report.  Table  6  shows  the  response 
patterns  and  NCI  and  ICI  values  for  three  students.  Student  1  performed 
all  addition  problems  correctly  but  he  failed  to  change  the  sign  of 
the  subtrahend  when  he  converted  subtraction  problems  to  addition  problems. 
Student  2  always  added  the  two  numbers  and  took  the  sign  of  larger  number 
for  her  answers.  She  failed  to  discriminate  subtraction  problems  from 
addition  problems  and  applied  this  erroneous  rule  consistently  to  all 
16  tasks.  Student  3  achieved  fairly  well  but  he  occasionally  mistyped 
or  made  careless  mistakes.  Determining  the  rules  of  operation,  both 
right  and  wrong,  Is  discussed  In  detail  in  the  technical  report  80-2 
(Tatsuoka  et  al.,  I98O). 


Table  6 

The  response  patterns,  NCI,  iCI  and  scores 
of  three  students. 


Responses  to  four  parallel  forms  within 
the  64- 1  tern  test 


Student 


ICI 

Score 


Student  1 

Student  2 

nil  (+10) 

1111  (10) 

nil  (-2) 

0000  (-10) 

nil  (9) 

0000  (+15) 

1011  (9) 

0000  (+15) 

nil  (-19) 

nil  (-19) 

nil  (-2) 

0000  (-8) 

1111  (-14) 

nil  (-12) 

0000  (+14) 

0000  (+14) 

0000  (-23) 

0000  (-23) 

0000  (+13) 

0000  (+13) 

0000  (+9) 

0000  (+15) 

0000  (-14) 

0000  (-14) 

0000  (+2) 

nil  (+16)  ; 

0000  (-9) 

0000  (-11) 

0000  (+2) 

0000  (+16) 

0000  (-9) 

nil  (-15) 

0.9759 

wm 

1.0000 

1.0000 

27 

20 

0.7073 

.9268 

58 


The  tasks  are  ordered  by  their  overall  difficulties  over  four 
parallel  forms. 
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Table  6(a)  is  the  2x2  contingency  table  with  .90  as  the  dividing 
point  for  iCi  and  ,60  for  NCi  in  the  subsample  of  75  students  who  earned 
scores  of  53  or  higher,  whiie  Tabie  6(b)  is  the  corresponding  tabie  for 
47  students  with  scores  of  52  or  lower.  The  dividing  point,  52,  for 
scores  was  chosen  because  —  as  shown  in  Figure  1  --  12  out  of  each 
subtest  of  16  items  could  conceivably  be  answered  by  consistent  use  of 
an  erroneous  rule;  13  is  the  smallest  number  of  items  that  cannot  be  got 
correct  in  this  way,  which  corresponds  to  52  out  of  the  entire  test  of 
four  parallel  subtests  of  16  items  each.  Hence  it  seems  reasonable  to 
regard  52  or  less  as  "low  scores". 

Tabie  7 

Two-way  classifications  based  on  ICI  >  or  <  .90 
and  NCI  >  or  <  .60  among  students  with 
(a)  scores  >  53;  (b)  scores  <  52. 


ncH^I.^ 

.90 

1 

•  90 

Ncr-l£L 

.90 

.90 

.60 

8 

18 

26  .60 

3 

27 

30 

.60 

26 

23 

49  .60 

12 

5 

17 

34  41  75 

15 

32 

47 

30 


first  that,  despite  the  very  high  cut-off  point  of  .90  for  the  "high 
ICI"  category,  substantially  more  than  one-half  (73  out  of  122)  of  the 
students  have  high  ICI  values.  This  reflects  the  fact  that  the  examinees 
were  eighth  graders  who  had  already  received  fairly  extensive  Instruction 
In  signed-number  operations  and  hence  a  relatively  large  proportion  of  them 
showed  stable  response  patterns  over  the  four  parallel  subtests:  they 
had  already  approached  mastery  or  learning  plateaus  —  the  latter  being 
more  likely  in  this  case  in  view  of  the  fact  that  only  75  (or  61.5%)  of 
them  had  scores  over  52  out  of  64.  As  expected,  very  few  students 
(il  out  of  122)  had  low  ids  combined  with  high  NCIs.  Many  more  had 
low-ICI  ,  low-NCI  combinations;  these  are  students  who  made  more  or  less 
"random"  (or  at  least  non-systematic)  errors  but  who  nevertheless  made 
relatively  more  errors  among  items  that  were  easy  for  the  group  as  a 
whole.  It  is  reasonable  that  about  70%  of  the  low  scorers  who  had 
low  NCIs  fell  In  this  category,  while  only  53%  of  the  high  scorers  with 
low  NCIs  did. 

Returning  to  the  high- 1C  I  group,  members  of  which  the  three  students 
represented  in  Table  6  are  different  kinds  of  examples,  the  high-ICI, 
high-NCI  students  with  high  scores  are  the  "problem-free"  types  exemplified 
by  Student  3>  Unfortunately  there  are  only  18  such  students  while  there 
are  27  high-ICI,  hIgh-NCI  students  with  low  scores.  Student  1  is  an 
example  of  this  type  of  student,  and  his  response  patterns  corroborate 
the  characterization  in  Table  5,  that  he  has  a  serious  misconception,  but 
one  which  leads  to  correct  answers  (except  for  one  probably  careless 
error)  to  the  easy  items  (addition)  and  always  to  wrong  answers  to  the 
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hard  items.  In  the  hIgh-ICl,  low-MCI  category,  most  students  (23  out 
of  28)  are  high  scorers.  These  students  can  easily  be  "remediated", 
for  they  are  probably  getting  only  a  few  easy  items  wrong.  (It  is  an 
unfortunate  fact  that  a  few  easy  items  missed  can  cause  the  NCI  to 
become  quite  low.)  The  students  who  have  the  most  serious  problems 
are  the  high-ICI,  low-NCI  low  scorers,  of  whom  there  are  fortunately 
only  five.  Student  2  In  Table  ^  exemplifies  this  type,  and  her 
unusual  response  pattern  (which  remains  perfectly  consistent  over 
the  four  parallel  subtests)  will  take  quite  a  bit  of  remedial  instruction 
to  rectify. 
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APPLICATION  TO  ERROR  ANALYSIS:  II 

In  the  previous  section.  It  was  shown  how  the  existence  and 
seriousness  of  a  student's  misconception(s)  could  be  determined  by 
examining  the  ICI -NCI -score  combinations  with  each  quantity  dichotomized 
as  high/low.  Another  concern  we  have  in  error  analysis  is  to  discover 
the  extent  to  which  a  dataset  has  been  "contaminated"  —  in  the  sense  of 
its  having  its  unidimensionality  destroyed  —  by  the  consistent  occurrence 
of  erroneous  rules  of  operation.  This  can  be  done  by  drawing  a  scatter- 
plot  of  NCI  against  total  score  for  the  given  dataset  and  comparing  it 
with  the  corresponding  scatterplot  based  on  synthetic  data  generated  by 
the  72  erroneous  rules  referred  to  at  the  beginning  of  the  previous 
section.  It  is  convenient  also  to  draw  the  regression  line  of  NCI  on 
score  for  the  synthetic  data. 

The  procedure  Is  illustrated  in  Figjre  2,  where  the  scatterplot  of 
NCI  against  "task-mastery  score*"  for  the  November  data  (points 
represented  by  X's)  Is  superimposed  on  the  corresponding  scatterplot  for 
the  artiflcaily  produced  data  generated  by  erroneous  rules  (points 
represented  by  o's)  with  the  regression  line  shown.  It  can  be  seen  at  a 
glance  that  almost  ail  of  the  real  data  points  fall  above  the  regression 
line  for  the  synthetic  data.  In  fact,  a  large  majority  of  them  even 
fall  above  the  dashed  line  parallel  to  the  regression  line,  which  is 
drawn  one  S.E.  above  the  latter. 


*The  "task-mastery  score"  Is  defined  as  follows:  If  a  student  gets 
at  least  three  of  the  four  parallel  items  testing  a  given  task,  his/her 
mastery  score  for  that  task  is  1;  otherwise  it  is  0. 


We  may  conclude  from  the  foregoing  graphical  inspection  that  the 
November  data  were  fairly  "clean".  By  contrast,  the  data  obtained  from 
the  seventh  graders  tested  after  being  taught  signed-number  operations 
via  two  PLATO  sessions  of  about  one  hour  each  (see  Example  5  above) 
were  found  to  be  highly  "contaminated".  Figure  3  shows  the  scatterplot 
of  NCI  against  test  score  for  one  of  these  datasets,  in  which  many 
points  fell  in  the  region  occupied  by  the  synthetic  datapoints  in 
Figure  2,  suggesting  that  uses  of  erroneous  rules  abounded. 


Figure  2,  Scatterplot  of  NCi  vs,  total  score  for  the 
November  mastery  score  data,  superimposed  on  that  for 
72  synthetic  response  patterns  generated  by  erroneous  rules. 

X  =  real  data  (N  «»  127) 
o  ■  synthetic  data  (N  ■  72) 
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EXTRACTING  UNID»HENSIONAL  SUBSETS 
It^  Characteristic  Curve  (ICC)  theories  are  useful  and  powerful 
test-theory  models  especially  for  applications  in  adaptive  testing.  If 
the  test  items  are  drawn  from  a  single,  unidimens ionai  domain,  logistic 
models  are  convenient  for  estimating  the  item-curve  parameters.  Tatsuoka 
(1980)  examined  the  response  patterns  of  students  for  whom  ability 
estimates  with  known  item  parameters  failed  to  converge  by  the  maximum- 
likelihood  method,  and  found  them  all  to  have  low  NCI  values.  Conversely, 
when  the  NCI  Is  very  small,  the  max i mum- 1 i ke 1 i hood  method  often  fails 
to  yield  a  convergent  estimate  for  the  ability  variable  0.  Table  8  shows 
three  such  response  patterns  (along  with  one  for  which  the  0  estimate  did 
converge)  for  the  48- item  Stanford  Vocabulary  Test  taken  by  the  seventh 
graders  in  the  January  experiment.  The  item  parameters  for  the  three- 
parameter  logistic  model  fitted  to  these  data  were  estimated  by  LOG  I  ST 
(Wood,  Wingersky  6  Lord,  1976). 

Table  8 

One  Convergent  and  Three  Nonconvergent  Response  Patterns 
For  Estimating  0  by  the  MXL  Method  and  Their  NCI  Values 


Response  Patterns 

(48  items  ordered  roughly  from  easier  to  harder) 

esti¬ 

mated 

0 

NCI 

No.  of 
itera¬ 
tions 

7 

00001011 00000000 100011 000000000 1 0000 1 00000001 00 1 

-3.789 

.0158 

25® 

1 1 1100001000000000000000000100010000100110000000 

-1.335 

.2526 

7 

4 

000000000000000000000000000000000000001 111111111 

-4.957 

-1.0 

25 

5 

00000000000000000000000000001 111111111 0000000000 

-25.00 

-.4739 

25 

a  Iterations  were  terminated  at  25  tentatively,  but  the  decrements  of  three 
cases  exceeded  .001. 

b  These  two  response  patterns  are  taken  from  the  real  data  and  the  other 
two  are  hypothetical  response  patterns. 
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The  problem  of  non-convergence  of  max I mum- 1  ike 11  hood  estimation 
procedures  for  ICC  models  due  to  failure  of  the  data  to  exhibit  uni- 
dimensionality  has  been  plaguing  researchers  for  a  long  time.  Mokken 
(1970}  goes  as  far  as  to  state  that,  although  ICC  theory  has  many 
valuable  features,  studies  in  sociology  and  political  science  are  not 
quite  ready  to  take  advantage  of  the  refined  parameter  estimation 
methods  that  It  offers.  To  cope  with  this  situation,  he  deveioped 
a  techinque  for  extracting  scalable  subsets  of  Items  from  a  given 
dataset.  Similar  techniques  have  been  developed  in  the  fields  of 
educational  and  psychological  testing  (Krus.  1977;  Reynolds.  1976; 
Yamamoto  &  Wise.  1980).  Theoretically,  all  these  methods  (which  are 
based  on  order  analysis)  show  a  duality  between  items  and  examinees, 
and  hence  can  in  principle  be  used  for  extracting  subsets  of  examinees 
as  well  as  items  in  which  unidimensionality  will  hold.  In  practice, 
however,  most  if  not  all  of  them  would  be  quite  Ineficlent  for  extracting 
examinee  subsets  because  the  dominance  matrix  for  this  purpose  would 
be  of  order  equal  to  the  number  of  examinees  instead  of  test  items, 
and  would  thus  be  very  large.  We  therefore  present  a  new  technique, 
based  on  the  NCI,  for  extracting  unidimensional  examinee  subsets  that 
does  not  require  the  use  of  a  dominance  matrix  as  the  starting  point 
and  hence  is  probably  more  efficient  than  those  that  do  (although  we  have 
not  yet  made  a  formai  comparison).  Our  technique  is,  of  course,  just 
as  appilcabie  for  extracting  unidimensionai  item  subsets,  but  in  that 
case  its  advantage,  if  any,  over  previous  methods  is  probably  negligible. 
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Recalling  that  Cliff's  group  consistency  index  is  a 
weighted  average  of  the  NCI's  of  the  members  of  the  group  (with  the 
group  itself  defining  the  item-difficulty  order),  it  is  clear  that 
individuals  with  negative  NCI  values  are  detrimental  to  the  goal  of 
getting  a  large  C^^  value.  We  therefore  remove  any  individual  with  a 
negative  NCI  from  the  group  at  the  outset,  before  starting  our  extraction 
procedures.  Let  the  NCI  for  the  j-th  member  of  the  group  thus  purged  be 

2U. 

NCI  .  =  — -  1 

-»  U. 

J 

Then,  by  Property  the  overall  consistency  of  the  group,  whose  size 
we  denote  by  N,  is 


N 

C  (SJ  -  E  w,  NCI. 

P  N  j  j 


9 


where 


w. 

J 


=  U./ 
J 


N 

E  U. 
i=i  ^ 


Now  suppose  we  remove  the  k-th  member  of  the  group  and  denote  the 
reduced  data  matrix  by  Then  , 


*lt  is  realized  that  removal  of  an  individual  from  a  group  may  con¬ 
ceivably  alter  the  difficulty-order  of  the  Items,  and  hence  the  NCI's; 
but  this  is  improbable,  especially  when  the  deleted  individual  has  a 
small  NCI,  as  we  shall  presently  see  he/she  does. 


where 


N 

“  2w;mcI. 
P  N-1  j  j 


w, 


U  /  Z  U. 
J  jtk  J 


N 

U  /(  Z  U  -  U.  )  . 
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The  resulting  change  in  overall  consistency  is 


-  Z  wtNCI,  -  Z  w.NCI . 
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N  N 
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N 
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where  U.  is  an  abbreviation  for  £  U.. 

j-1 

Therefore 

('»)  AC  =  - = -  £  w.MCI.  -  wCNCI. 

P  U.  -  j-1  J  J  k  k 


^[vv-“v  . 
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Ue  thus  see  that,  in  order  to  make  AC^  as  large  as  possible  —  that  is, 
to  have  the  removal  of  the  k-th  member  result  in  as  large  an  increase 
in  Cp  as  possible  —  two  conditions  must  be  satisfied;  namely,  should 

be  as  small  as  possible,  while  U|^  should  be  as  large  as  possible.  The 
first  of  these  conditions  Is  intuitively  obvious.  Since  the  overall 
Cp  is  a  weighted  average  of  the  individual  NCI's,  elimination  of  the 
smallest  NCI  would  be  expected  to  increase  the  group  Cp  the  most. 

However,  since  the  latter  is  a  weighted  rather  than  a  straight  average 
of  the  NCI's,  it  is  also  necessary  that  the  NCI  to  be  eliminated  have 
as  high  a  weight  as  possible,  namely,  that  the  associated  U|^  be  as 
large  as  possible.  Recalling  that 


it  follows  that  X|^  should  be  as  close  as  possible  to  n/2  in  order  for 
U|^  to  be  large. 


From  a  purely  mathematical  standpoint,  the  above  optimizing  condi¬ 
tion  for  ACp  would  require  our  actually  computing  AC^  for  each  potential 
individual  to  be  removed,  for  there  could  be  a  tradeoff  between  NCI|^ 
being  small  and  U|^  being  large  (i«e.,  |X|^  -  n/2|  being  small).  In 
practice,  however.  It  Is  highly  unlikely  that  a  person  with  a  small 
NCI  would  have  a  middling  score  that  could  yield  a  large  U|^.  That  is, 
practically  everyone  with  a  small  NCI  would  have  a  relatively  extreme 
score,  leading  to  a  fairly  small  U|^.  Hence,  the  smallness  of  NCI 
becomes  the  overriding  consideration  In  selecting  the  individual  to 
be  removed.  It  therefore  suffices  to  compute  AC^  in  accordance  with 
Equation  (4)  for  Just  those  members  of  the  group  who  have  the  few 
smallest  values  of  NCI  and  select  the  one  among  them  that  yields  the 
largest  AC^. 

In  the  foregoing  manner,  we  would  successively  remove  the  member 
of  the  remaining  group  that  produces  the  largest  increase  in  the  overall 
C^  (noting  that  U.  and  l*ave  to  be  recomputed  at  each  step), 

until  the  value  of  C  achieves  a  satisfactory  target  magnitude. 


DISCUSSION 
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The  subset  of  examinees  (or  Items)  extractable  by  the  technique 
described  In  the  preceding  section  —  or,  indeed,  by  all  earlier 
methods,  to  our  knowledge  —  is  uni  dimensional  (or  nearly  so)  in 
the  scalability  or  order-theoretic  sense.  On  the  other  hand,  the 
unidimensionality  of  data  required  for  satisfactory  practical 
functioning  of,  and  parameter  estimation  In,  ICC  models  Is  more 
closely  related  to  that  in  the  factor-analytic  sense.  Since  it  is 
well  known  (e.g.,  Guttman,  1950;  DuBois,  1970)  that  scalability  of 
a  set  of  items  leads  to  a  simplex  and  —  depending  on  the  distribution 
of  difficulties  of  the  items  —  may  produce  a  correlation  matrix  with 
up  to  n/2  factors,  it  may  seem  meaningless  to  strive  for  uni  dimen¬ 
sionality  in  the  sen  e  of  scalability  when  the  purpose  is  to  improve  the 
applicability  of  ICC  models. 

Despite  the  foregoing  circumstance,  experience  has  shown  that 
when  a  set  of  items  approximates  scalability  in  a  given  group  of 
examinees,  the  factorial  structure  also  becomes  "cleaner"  and  the 
estimability  of  the  latent  trait  parameter  is  improved.  This  is 
described  in  detail  by  Birenbaum  and  Tatsuoka  (1980),  who  found  that 
all  these  Improvements  —  scalability,  factorial  determinancy  and 
estimability  of  6  —  result  siimiltaneously  when  item  scores  are 


’''Lord  and  Novick  (1968,  pp.  37^“37$)»  show  that  the  matrix  of  item 
tetrachonics  having  unit  rank  (when  communal  I  ties  are  inserted  in  the 
diagonal)  is  a  sufficient  but  "very  far  from  being  (a)  necessary" 
condition  for  the  assumption  of  local  independence  of  the  items  to  hold. 


modified  by  assigning  zeros  to  those  items  that  were  deemed  (by  an 
elaborate  system  of  error  analsyis)  to  be  correctly  answered  for 
wrong  reasons.  Thus,  improving  unidimensionality  in  the  scalability 
sense  --  or.  to  put  It  another  way.  removing  examinees  with  aberrant 
response  patterns  --  does  enhance  the  practical  applicability  of  ICC 
models  up  to  a  certain  point.  But  there  are  limits  to  the  efficacy 
of  this  approach,  which  are  discussed  elsewhere  (Tatsuoka  &  Tatsuoka, 
1980). 

We  now  turn  our  attention  to  a  couple  of  difficulties  with  the  NCI 
that  we  have  yet  to  resolve  to  our  satisfaction.  The  first  is  the 
excessively  small  (l.e..  close  to  -1)  value  received  by  a  student 
whose  test  score  would  have  been  perfect  except  for  his/her  getting 
one  or  two  very  easy  items  wrong  by  mistyping  or  some  other  clerical 
error.  For  instance,  consider  the  response  pattern  (111111111101). 
whose  NCI  value  is  -.818.  Yet.  this  students  getting  the  second 
easiest  item  wrong  is  almost  certainly  due  to  a  random  clerical  error, 
and  hence  the  response  pattern  should  not  be  regarded  as  "extremely 
atypical"  in  the  sense  of  its  implying  a  serious  misconception.  In 
particular,  it  seems  incongruous  that  such  a  response  pattern  should 
be  automatically  deleted  from  the  outset  in  the  method  for  extraction 
of  unidimensional  subsets  discussed  in  the  preceding  section.  Thus, 
this  extreme  sensitivity  of  the  NCI  to  one  or  two  "happens tantia I" 
zeros  in  a  response  pattern  that  would  otherwise  receive  a  value  of 
■fl.O  Is  an  undesirable  property  so  long  as  we  adopt  overall  group 
consistency  as  the  criterion  for  extracting  unidimensional  subsets. 
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Fortunately,  however,  the  above  defect  of  the  NCI  does  not  affect 
its  usefulness  in  the  diagnostic  procedure  utilizing  the  fourfold 
table,  based  on  the  NCI,  ICI  and  score  combination,  displayed  in  Table  4. 
It  will  be  recalled  that,  so  long  as  the  total  score  is  high,  a  student 
with  low  NCI  will  not  be  diagnosed  as  having  serious  problems  even 
when  the  ICI  Is  high.  It  is  also  seen  from  Table  4  that,  before  any 
diagnosis  can  be  made  on  the  basis  of  the  NCI's  being  high  or  low,  it 
is  essential  to  examine  whether  the  total  test  score  is  high  or  low. 

All  in  ali,  it  appears  that  the  ICI  is  the  more  useful  of  the 
two  Indices  for  diagnostic  purposes.  Its  drawback  is  that  it 
requires  the  test  to  be  constructed  out  of  two  or  more  parallel  subtests. 
Alternatively,  we  might  say  that,  for  achievement  tests  to  perform  as 
powerful  diagnostic  tools,  they  should  incorporate  several  parallel 
subparts. 

It  was  pointed  out  earlier  that  Equation  (2)  fails  when 
the  test  score  is  either  zero  or  perfect,  making  U  ■  0.  Intuitively, 

Cp  should  have  the  value  1  In  both  these  cases,  since  a  response  vector 
with  all  elements  equal  0  and  one  with  all  elements  equal  1  are  both 
Guttman  vectors.  This  can  be  brought  about  in  one  of  two  ways: 

(a)  by  arbitrarily  defining  C^  •  1  when  U  -  0  despite  the  fact  that 
Equation  (2)  does  not  apply  in  this  case  — *  much  as  we  define  01  ■  I 
even  though  the  definition  nl  -  1«2<3”*  n  does  not  make  sense  for 
nH);  or  (b)  by  changing  the  very  definition  of  C^  to  read 


(5) 


1 


2tU  +1) 


kk 


instead  of  C  ■  2U  /U  -  1  as  in  Equation  (2). 

P  ® 

Alternative  (a)  has  the  advantage  of  preserving  the  definition 
of  Cliff's  consistency  index  as  a  weighted  average 

N 

E  w.  NCI . 

j-1  J  J 


with 

N 

w,  »  U./  Z  Ui 

J  J  i.i 


as  stated  in  Equation  (3).  For  any  pattern  with  Uj  -  0,  Wj  would  be 
0  and  hence  the  value  NCIj  -  1  would  not  enter  into  the  weighted 
average  for  computing  C^^,  This  is  consistent  with  Cliff's  definition, 
because  perfect  and  ail-zero  response  patterns  do  not  contribute 
anything  to  a  dominance  matrix,  since  no  item  dominates  any  other  item 
in  such  response  patterns. 

On  the  other  hand,  alternative  (a)  has  the  disadvantage  of  rendering 

undefined  the  ICI  for  a  student  with  perfect  (all  all-zero)  response 

patterns  on  all  of  the  parallel  subtests,  for  the  combining  weights 

for  all  the  (individual ly  perfect)  C  's  would  then  take  the  indeterminate 

P 

form  0/0,  Thus,  it  would  require  another  definition  by  fiat  to  give 
such  a  student's  ICI  the  value  of  1,  which  is  what  is  should  be, 
since  all  the  response  patterns  are  identical. 
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By  contrast,  alternative  (b)  would  avoid  this  difficulty.  With 
the  revised  definition  (5)  for  the  combining  weight  associated  with 
the  of  the  J-th  among  a  set  of  m  response  patterns  would  be 


16) 


w. 


m 

I  U  +m 
j-1 


Consequently,  the  combining  weights  used  in  calculating  the  ICI  for 
a  student  who  consistently  shows  perfect  (or  ail-zero)  response 
patterns  on  m  parallel  subsets  would  uniformly  equal  Wj  »  1/m.  Hence 
his/her  ICI  will  now  be  1.0,  as  it  should  be.  On  the  other  hand, 
alternative  (a)  would  lead  to  an  overall  group  consistency  index  that 
does  not  agree  with  Cliff's  C^^  since  each  NCI  would  no  longer  be 
a  linear  transform  of  Ua/U. 

Thus,  each  of  the  alternatives  for  making  NCI  take  the  value  1.0 
for  perfect  and  all-zero  respqnse  patterns  has  its  advantage  and  its 
disadvantage,  and  we  have  a  dilemma.  In  view  of  the  more  important 
role  played  by  the  ICI  compared  to  the  NCI  in  error  diagnosis,  we  are 
inclined  to  favor  alternative  (b).  However,  further  investigation  of 
other  possible  Implications  carried  by  definition  ($)  for  pattern 
conformity  needs  to  bemade  before  we  mqke  a  definite  commitment. 
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